Skip to content

Enforce in bootstrap that check must have stage at least 1 #143048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

Kobzol
Copy link
Member

@Kobzol Kobzol commented Jun 26, 2025

This PR is another step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917, this time dealing with x check.

It enforces the invariant that:

  • We check std stage N with rustc stage N
  • We check everything else stage N with rustc stage N - 1

It creates a single function that prepares a proper build compiler for checking something, and also adds snapshot tests for various common check steps. Some obsolete code was also removed.

The default check stage also becomes 1, for all profiles. I tested manually that x check std with download-ci-rustc still works and doesn't build rustc locally.

Subsumes #139170.

r? @ghost

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 26, 2025
@Kobzol Kobzol force-pushed the bootstrap-check-stage-1 branch from f06e41e to aed5f23 Compare June 26, 2025 08:03
@Kobzol
Copy link
Member Author

Kobzol commented Jun 26, 2025

CC @jieyouxu (one of many heaps of bootstrap fixup commits I have in store 😆)

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Collaborator

bors commented Jun 26, 2025

☔ The latest upstream changes (presumably #141899) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 26, 2025
@Kobzol Kobzol force-pushed the bootstrap-check-stage-1 branch from aed5f23 to a36ee24 Compare June 26, 2025 14:29
@Kobzol
Copy link
Member Author

Kobzol commented Jun 26, 2025

Cross-compilation checks are now doing too much work, I need to take a look at how we can reduce that, because x check --stage1 with cross-compilation is no longer a thing.

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Collaborator

bors commented Jul 1, 2025

☔ The latest upstream changes (presumably #143254) made this pull request unmergeable. Please resolve the merge conflicts.

@Kobzol Kobzol force-pushed the bootstrap-check-stage-1 branch from a36ee24 to 22fb548 Compare July 1, 2025 15:43
@Kobzol Kobzol marked this pull request as ready for review July 1, 2025 15:44
@rustbot rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 1, 2025
@rustbot
Copy link
Collaborator

rustbot commented Jul 1, 2025

This PR modifies src/bootstrap/defaults.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

@Kobzol
Copy link
Member Author

Kobzol commented Jul 1, 2025

r? @jieyouxu

@rustbot
Copy link
Collaborator

rustbot commented Jul 1, 2025

jieyouxu is not on the review rotation at the moment.
They may take a while to respond.

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Collaborator

bors commented Jul 1, 2025

☔ The latest upstream changes (presumably #143287) made this pull request unmergeable. Please resolve the merge conflicts.

@jieyouxu jieyouxu removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 2, 2025
Copy link
Member

@jieyouxu jieyouxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this a lot in general, it does feel much more self-consistent and coherent.

I left some questions/discussions for things that I wanted to double-check, but overall looks good!

Comment on lines 183 to 191
self.stage.or(self
.built_by
.map(|compiler| if self.name == "std" { compiler.stage } else { compiler.stage + 1 }))
Copy link
Member

@jieyouxu jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: this makes me feel a bit uneasy, but I guess the logic here is: anything that's not the standard library should be considered a non-library artifact, and should be considered a product of the stage n (compiler, standard library) pair (and so gets numbered as the next stage)? I.e. reading back https://hackmd.io/QXj6LqntQTKPgm3uBuE-aA again, the core model is

stage corresponds to what gets built, not what is used to build it

(The following is me trying to make sure I understand this)


Let's call the (compiler, library) pair12

$$\mathbf{CLP}_{\mathrm{stage}} := \left\langle \text{compiler}_{\mathrm{stage}}, \text{library}_{\mathrm{stage}} \right\rangle$$

where the separation between stages is delineated by which $\mathrm{compiler}$ gets built. Then, by design, because the standard library is "special" and tightly coupled with the compiler, we fix the immediate standard library built by $\mathrm{compiler}_{\mathrm{stage}}$ to be of the same $\mathrm{stage}$, i.e. $\mathrm{library}_{\mathrm{stage}}$.

With the stage 0 redesign, we now have a nice straightforward "derivation chain" ($\longrightarrow$; artifacts can be used to build new artifacts) property where

$$\left\langle \text{compiler}_{\mathrm{stage}}, \text{library}_{\mathrm{stage}} \right\rangle \longrightarrow \left\langle \text{compiler}_{\mathrm{stage + 1}}, \text{library}_{\mathrm{stage + 1}} \right\rangle$$

actually holds even for $\mathrm{stage} = 0$.

However, for:

  • bootstrap tools
    Tools that are unconditionally built by the stage 0 $\mathbf{CLP}$, i.e. always depends on $\mathbf{CLP}_{0}$. Their stage numbering will be called $\mathrm{BootstrapTool}_{1}$, at least in the short-term.
  • host rustc tools
    Tools that depend on rustc_private and so on $\mathrm{compiler}_{\mathrm{stage}}$. This means that they will be numbered as $\mathrm{HostRustcTool}_{\mathrm{stage} + 1}$.
  • std tools
    Tools that depend on the whole $\mathbf{CLP}_{\mathrm{stage}}$, i.e. both $\mathrm{compiler}_{\mathrm{stage}}$, and $\mathrm{library}_{\mathrm{stage}}$. Similiarly, this means that they will be numbered as $\mathrm{StdTool}_{\mathrm{stage} + 1}$.

They then follow the general "what gets built" model (?).

mental-model

Footnotes

  1. I can't come up with a better name, it's not exactly a toolchain, neither is it a "complete" sysroot I think

  2. Not to be confused with the Chilean Peso.

Comment on lines 255 to 256
builder.std(build_compiler, host);
builder.std(build_compiler, target);
builder.ensure(Std::new(build_compiler, target));
Copy link
Member

@jieyouxu jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: wait, why is this different? For $\mathrm{stage} \ge 1$, isn't this equivalent? I.e.

if compiler.stage == 0 {
if target != compiler.host {
panic!(
r"It is not possible to build the standard library for `{target}` using the stage0 compiler.
You have to build a stage1 compiler for `{}` first, and then use it to build a standard library for `{target}`.
",
compiler.host
)
}
// We still need to link the prebuilt standard library into the ephemeral stage0 sysroot
self.ensure(StdLink::from_std(Std::new(compiler, target), compiler));
} else {
// This step both compiles the std and links it into the compiler's sysroot.
// Yes, it's quite magical and side-effecty.. would be nice to refactor later.
self.ensure(Std::new(compiler, target));
}

At this point, wouldn't we have already rejected invoking ./x check flow with $\mathrm{stage} = 0$?

Copy link
Member Author

@Kobzol Kobzol Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Equivalent to what? :) I changed it just to this:

let build_compiler = builder.compiler(builder.top_stage, host);
builder.ensure(Std::new(build_compiler, target));
build_compiler

because I remembered that stdlib's proc macros and build scripts are always compiled using the stage0 libstd. We only need the host stdlib when compiling host code of other modes (yeah, I know..).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except! We actually need the host stdlib when checking the tool itself (not stdlib). Aaaaargh.

Copy link
Member

@jieyouxu jieyouxu Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Equivalent to what?

NGL, I don't know what I meant by this myself -- when I click at this comment, it's outdated against something, and I can't tell what it was...

@@ -48,7 +48,7 @@ ENV SCRIPT \
python3 ../x.py build src/tools/build-manifest && \
python3 ../x.py test --stage 0 src/tools/compiletest && \
python3 ../x.py check compiletest --set build.compiletest-use-stage0-libtest=true && \
python3 ../x.py check --stage 1 --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \
python3 ../x.py check --stage 2 --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: actually, can you add a comment for this specifically that we're exercising cross-compile here, so --stage 1 isn't sufficient?

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 3, 2025
…eyouxu

Add bootstrap check snapshot tests

Split off from rust-lang#143048, so that we get a baseline of how check behaved before we make changes to it. Note that the output of the check snapshot tests is suboptimal in many places, as we're missing information about stages and the build compiler. That will be changed in rust-lang#143048.

r? `@jieyouxu`
rust-timer added a commit that referenced this pull request Jul 3, 2025
Rollup merge of #143316 - Kobzol:bootstrap-check-tests, r=jieyouxu

Add bootstrap check snapshot tests

Split off from #143048, so that we get a baseline of how check behaved before we make changes to it. Note that the output of the check snapshot tests is suboptimal in many places, as we're missing information about stages and the build compiler. That will be changed in #143048.

r? `@jieyouxu`
@Kobzol Kobzol force-pushed the bootstrap-check-stage-1 branch from 22fb548 to fae5ba2 Compare July 3, 2025 15:59
@rust-log-analyzer
Copy link
Collaborator

The job pr-check-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)
#11 DONE 2.3s

#12 [ 5/11] RUN npm install [email protected] [email protected] [email protected] -g
#12 5.930 npm WARN deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
#12 5.948 npm WARN deprecated @types/[email protected]: This is a stub types definition. minimatch provides its own type definitions, so you do not need this installed.
#12 5.970 npm WARN deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
#12 5.981 npm WARN deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
#12 6.018 npm WARN deprecated @humanwhocodes/[email protected]: Use @eslint/config-array instead
#12 6.045 npm WARN deprecated [email protected]: Glob versions prior to v9 are no longer supported
#12 6.051 npm WARN deprecated @humanwhocodes/[email protected]: Use @eslint/object-schema instead
---
#16 3.030 Building wheels for collected packages: reuse
#16 3.031   Building wheel for reuse (pyproject.toml): started
#16 3.245   Building wheel for reuse (pyproject.toml): finished with status 'done'
#16 3.246   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132719 sha256=d2a2565e7037ad3883fb9337653f2e25bbb588534fbef3697286cbc26d1bf634
#16 3.247   Stored in directory: /tmp/pip-ephem-wheel-cache-hb5rgpkr/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#16 3.249 Successfully built reuse
#16 3.250 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#16 3.651 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#16 3.651 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#16 DONE 3.7s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants